Systems and methods for deblurring medical images using deep neural network

ABSTRACT

Methods and systems are provided for deblurring medical images using deep neural networks. In one embodiment, a method for deblurring a medical image comprises receiving a blurred medical image and one or more acquisition parameters corresponding to acquisition of the blurred medical image, incorporating the one or more acquisition parameters into a trained deep neural network, and mapping, by the trained deep neural network, the blurred medical image to a deblurred medical image. The deep neural network may thereby receive at least partial information regarding the type, extent, and/or spatial distribution of blurring in a blurred medical image, enabling the trained deep neural network to selectively deblur the received blurred medical image.

TECHNICAL FIELD

Embodiments of the subject matter disclosed herein relate to processingmedical images, such as magnetic resonance images (MM), computedtomography (CT) images, X-ray images, ultrasound images, etc., and moreparticularly, to reducing blurring in medical images using deep neuralnetworks.

BACKGROUND

Medical imaging systems such as magnetic resonance imaging (MM) systems,computed tomography (CT) systems, positron emission tomography (PET)systems, X-ray systems, ultrasound systems, etc., are widely used toobtain internal physiological information of a subject (e.g., apatient). Medical images obtained by these imaging modalities mayinclude one or more types of blurring which may be caused by transientsignal acquisition, tissue/patient specific effects, imaging equipmentproperties, and so on. Blurring may degrade the resolution anddiagnostic quality of medical images, and may further reduce theefficacy of downstream image processing methods which may have beentrained on sharp, unblurred medical images. Deep learning approacheshave been proposed for use in deblurring images, however the performanceof current deep learning approaches in deblurring is inconsistent, andoften does not produce a sufficient degree of deblurring. Therefore,exploring deep learning techniques to identify new ways for consistentlydeblurring medical images is generally desired.

SUMMARY

The present disclosure at least partially addresses the issues describedabove. In one embodiment, the present disclosure provides a method fordeblurring a medical image using deep neural networks. The methodcomprises, receiving a blurred medical image and one or more acquisitionparameters corresponding to acquisition of the blurred medical image,incorporating the one or more acquisition parameters into a trained deepneural network, and mapping, by the trained deep neural network, theblurred medical image to a deblurred medical image. By receiving both ablurred medical image and an acquisition parameter, wherein theacquisition parameter corresponds to one or more parameters of animaging system used during acquisition of the blurred medical image, adeep neural network may be at least partially informed of the type,extent, and/or distribution of blurring present in the blurred medicalimage, thereby enabling the deep neural network to map the blurredmedical image to a corresponding deblurred medical image with a greaterdegree of consistency, even when multiple types of blurring are presentin the medical image, or when the extent/type of blurring varies bypixel/voxel.

The above advantages and other advantages and features of the presentdescription will be readily apparent from the following DetailedDescription when taken alone or in connection with the accompanyingdrawings. It should be understood that the summary above is provided tointroduce in simplified form a selection of concepts that are furtherdescribed in the detailed description. It is not meant to identify keyor essential features of the claimed subject matter, the scope of whichis defined uniquely by the claims that follow the detailed description.Furthermore, the claimed subject matter is not limited toimplementations that solve any disadvantages noted above or in any partof this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of this disclosure may be better understood upon readingthe following detailed description and upon reference to the drawings inwhich:

FIG. 1 is a schematic diagram illustrating an image processing systemfor deblurring medical images using a deep neural network and anacquisition parameter transform, according to an exemplary embodiment;

FIG. 2 is a schematic diagram illustrating the layout of a firstembodiment of a deep neural network, and an acquisition parametertransform, which can be used in the image processing system of FIG. 1,according to an exemplary embodiment;

FIG. 3 is a schematic diagram illustrating the layout of a secondembodiment of a deep neural network, and an acquisition parametertransform, which can be used in the image processing system of FIG. 1,according to an exemplary embodiment;

FIG. 4 is a schematic diagram illustrating an architecture of a deepneural network which can be used in the system of FIG. 1, according toan exemplary embodiment;

FIG. 5 is a schematic diagram illustrating an acquisition parametertransform architecture which can be used in the system of FIG. 1,according to an exemplary embodiment;

FIG. 6 is a schematic diagram illustrating an acquisition parametertransform architecture which can be used in the system of FIG. 1,according to an exemplary embodiment;

FIG. 7 is a flow chart illustrating a method for deblurring a medicalimage using a deep neural network and an acquisition parametertransform, according to an exemplary embodiment;

FIG. 8 shows signal decay due to T2 and a corresponding point spreadfunction.

FIG. 9 shows a comparison between blurred and deblurred medical images,according to an exemplary embodiment.

The drawings illustrate specific aspects of the described systems andmethods for deblurring a medical image using a deep neural network andone or more acquisition parameter transforms. Together with thefollowing description, the drawings demonstrate and explain thestructures, methods, and principles described herein. In the drawings,the size of components may be exaggerated or otherwise modified forclarity. Well-known structures, materials, or operations are not shownor described in detail to avoid obscuring aspects of the describedcomponents, systems and methods.

DETAILED DESCRIPTION

Medical images are prone to blurring, which limits image resolution anddegrades diagnostic quality. In one example, blurring in magneticresonance (MR) images may result from signal modulation during atransient acquisition and/or from variable flip angles and/or relaxationeffects during a readout train. One example is T2 blurring in singleshot fast spin echo. In another example, in CT imaging, blurring mayresult as a function of detector size, detector number,source-to-detector distance, collimator geometry, slice thickness, dose,etc. In another example, in PET imaging, blurring may result as afunction of detector type, detector geometry, ring radius, positronrange (isotope type), depth/line of response, and voxel activity. In yetanother example, in ultrasound imaging, blurring may result as afunction of transducer (crystal and array) geometry, central frequency,ringdown, spatial pulse length, focal depth, beam apodization, side lobeamplitude, slice thickness, etc. Thus, blurring occurs in a wide rangeof medical imaging modalities, and may occur as a function of one ormore acquisition parameters, where, as used herein, acquisitionparameters will be understood to include one or more parameters of theimaging system, the imaged tissue, and/or environmental conditions atthe time of acquisition of a medical image (temperature, humidity,ambient magnetic field strength, etc.).

The process of deblurring medical images may be complicated when thetype/source of blurring is not known and/or when the extent of blurringvaries throughout the image. Take MR as an example. In single shot fastspin echo, blurring may result from the combination of low (and possiblyvariable) refocusing angles in combination with other factors, includingtissue specific relaxation times, view ordering, partial volume effects,B₁ field inhomogeneity, etc. In some cases, each voxel/pixel may have aunique blurring function. Deblurring is an ill-posed inverse problem,that is difficult to solve with conventional methods. Further,conventional methods have the drawback of noise amplification, resultingin sharper but noisier images.

The following description relates to various embodiments for deblurringmedical images using deep neural networks, which may at least partiallyaddress the above identified issues. In particular, a medical image isacquired by an imaging system with one or more acquisition parametersapplied during the acquisition of the medical image. Acquisitionparameters may include various imaging system settings used to acquirethe medical image. For example, acquisition parameters for an MR imagemay include one or more of an echo-train-length, a repetition time (TR),an echo time (TE), flip angle, inversion time, etc. Acquisitionparameters may further include one or more of a dimension of the medicalimage, voxel spatial dimensions (the volume of space represented by eachvoxel in a medical image), sampling pattern, acquisition order,acceleration factor, fat saturation setting (ON or OFF), B₀ shim modeselection, RF drive mode selection, physiological signals, physiologicalstate, image reconstruction parameters, and so on. The acquired medicalimage, which may include blurring, is then processed using a traineddeep neural network and one or more acquisition parameter transforms.The one or more acquisition parameters applied during the acquisition ofthe medical image are input into the acquisition parameter transform,and the output therefrom may be incorporated into the deep neuralnetwork in various ways. By incorporating the output of the acquisitionparameter transform into the deep neural network, information regardingthe type, extent, and distribution of blurring within the medical imagemay be determined by the deep neural network and used to deblur themedical image in a more consistent manner, without increasing noisewithin the medical image.

In some embodiments, the output of the acquisition parameter transformmay comprise a point-spread-function based on one or more acquisitionparameters, wherein a deconvolution kernel and/or convolution kernel foruse in the deep neural network may be based on the point-spread-functiondetermined by the acquisition parameter transform. In this way,deblurred/sharp medical images may be produced from correspondingblurred medical images. Further, the deep neural network and acquisitionparameter transform (in embodiments in which the acquisition parametertransform comprises a neural network) may be trained before being put inuse.

As used herein, deblurring is the process of removing blurring artifactsfrom images, such as blurring caused by defocus aberration or motionblurring, or other types of blurring artifacts. Conventionally, blur maybe modeled as the convolution of a (space-varying and/or time-varying)point-spread-function over a hypothetical sharp input image, where boththe sharp input image and the point-spread-function may be unknown. Assuch, deblurring comprises the process of at least partially reversingor mitigating the convolution of the sharp image by one or morepoint-spread-functions, to obtain or approximate the sharp input image,and is therefore referred to as an inverse problem. Sharp images (orsharp medical images) may therefore be produced by deblurring a blurredimage (or blurred medical image), and conversely, blurred images may beproduced by introducing one or more blurring artifacts into a sharpimage. It will be appreciated that in some conditions, sharp medicalimages may be acquired directly using an imaging system.

Referring to FIG. 1, a medical image processing system 100 is shown, inaccordance with an exemplary embodiment. In some embodiments, themedical image processing system 100 is incorporated into a medicalimaging system, for example, an MRI system, CT system, X-ray system, PETsystem, ultrasound system, etc. In some embodiments, the medical imageprocessing system 100 is disposed at a device (e.g., edge device,server, etc.) communicably coupled to the medical imaging system viawired and/or wireless connections. In some embodiments, the medicalimage processing system 100 is disposed at a separate device (e.g., aworkstation) which can receive images from the medical imaging system orfrom a storage device which stores the images generated by the medicalimaging system. The medical image processing system 100 may compriseimage processing system 31, user input device 32, and display device 33.

Image processing system 31 includes a processor 204 configured toexecute machine readable instructions stored in non-transitory memory206. Processor 204 may be single core or multi-core, and the programsexecuted thereon may be configured for parallel or distributedprocessing. In some embodiments, the processor 204 may optionallyinclude individual components that are distributed throughout two ormore devices, which may be remotely located and/or configured forcoordinated processing. In some embodiments, one or more aspects of theprocessor 204 may be virtualized and executed by remotely-accessiblenetworked computing devices configured in a cloud computingconfiguration.

Non-transitory memory 206 may store deep neural network module 208,acquisition parameter transform module 210, training module 212, andmedical image data 214. Deep neural network module 208 may include oneor more deep neural networks, comprising a plurality of parameters(including weights, biases, activation functions), and instructions forimplementing the one or more deep neural networks to receive blurredmedical images and map the blurred medical image(s) to output, wherein adeblurred medical image corresponding to the blurred medical image maybe produced from the output. For example, deep neural network module 208may store instructions for implementing a neural network, such as theconvolutional neural network (CNN) of CNN architecture 400, shown inFIG. 4. Deep neural network module 208 may include trained and/oruntrained neural networks and may further include various data, ormetadata pertaining to the one or more neural networks stored therein.

Non-transitory memory 206 also stores acquisition parameter transformmodule 210, wherein one or more trained and/or untrained acquisitionparameter transforms, and associated data and/or metadata, may bestored. Acquisition parameter transforms may be configured to mapacquisition parameters to output, wherein the output may be incorporatedinto one or more deep neural networks of deep neural network module 208.In some embodiments, an acquisition parameter transform may beconfigured to predict one or more point spread functions based on one ormore input acquisition parameters. In some embodiments, acquisitionparameter transforms may comprise analytical functions or models,wherein the function/model receives one or more acquisition parametersas model arguments, and produces an output using the functional mappingfrom the acquisition parameters. In one example, acquisition parametertransforms may include Fourier transforms for mapping MRI echo sequencesto intensity point-spread-functions.

Non-transitory memory 206 may further store training module 212, whichcomprises instructions for training one or more of the deep neuralnetworks stored in deep neural network module 208 and/or acquisitionparameter transforms stored in acquisition parameter transform module210. Training module 212 may include instructions that, when executed byprocessor 204, cause image processing system 31 to conduct one or moreof the steps of method 700, discussed in more detail below. In someembodiments, training module 212 includes instructions for implementingone or more gradient descent algorithms, applying one or more lossfunctions, and/or training routines, for use in adjusting parameters ofone or more deep neural networks of deep neural network module 208and/or acquisition parameter transforms of acquisition parametertransform module 210. In some embodiments, training module 212 includesinstructions for intelligently selecting training data pairs frommedical image data 214. In some embodiments, training data pairscomprise corresponding pairs of blurred and deblurred/sharp medicalimages of a same anatomical region. In some embodiments, training module212 includes instructions for generating training data pairs byapplying/adding one or more blurring artifacts to sharp medical imagesto produce a blurred medical image. In some embodiments, the trainingmodule 212 is not disposed at the image processing system 31. The deepneural network module 208 includes trained and validated network(s).

Non-transitory memory 206 further stores medical image data 214. Medicalimage data 214 includes for example, MR images captured from an MRIsystem, ultrasound images acquired by an ultrasound system, etc. Forexample, the medical image data 214 may store blurred and/or sharpmedical images. In some embodiments, medical image data 214 may includea plurality of training data pairs comprising pairs of blurred and sharpmedical images. In some embodiments, medical image data may include oneor more blurring point spread functions used to produce a blurredmedical image from a sharp medical image.

In some embodiments, the non-transitory memory 206 may includecomponents disposed at two or more devices, which may be remotelylocated and/or configured for coordinated processing. In someembodiments, one or more aspects of the non-transitory memory 206 mayinclude remotely-accessible networked storage devices configured in acloud computing configuration.

Image processing system 100 may further include user input device 32.User input device 32 may comprise one or more of a touchscreen, akeyboard, a mouse, a trackpad, a motion sensing camera, or other deviceconfigured to enable a user to interact with and manipulate data withinimage processing system 31. As an example, user input device 32 mayenable a user to make a selection of a medical image to performdeblurring on.

Display device 33 may include one or more display devices utilizingvirtually any type of technology. In some embodiments, display device 33may comprise a computer monitor, and may display unprocessed andprocessed MR images and/or parametric maps. Display device 33 may becombined with processor 204, non-transitory memory 206, and/or userinput device 32 in a shared enclosure, or may be peripheral displaydevices and may comprise a monitor, touchscreen, projector, or otherdisplay device known in the art, which may enable a user to view medicalimages, and/or interact with various data stored in non-transitorymemory 206.

It should be understood that image processing system 100 shown in FIG. 1is for illustration, not for limitation. Another appropriate imageprocessing system may include more, fewer, or different components.

Turning to FIG. 2, a schematic of a first embodiment of a deblurringprocess 200 for deblurring a medical image is shown. Deblurring process200 may be implemented by image processing system 100 to at leastpartially remove one or more blurring artifacts from a blurred medicalimage. Deblurring system 200 serves to illustrate an embodiment of adeblurring system, wherein acquisition parameters 302 may be mapped viaacquisition parameter transform 304 to output, which is thenincorporated into deep neural network 324 via input layers 322 and usedto remove one or more blurring artifacts from blurred image 310. Bymapping acquisition parameters to output, and incorporating the outputinto deep neural network 324, the deep neural network 324 may receive atleast partial information regarding the extent, type, and distributionof blurring artifacts in blurred image 310, as the blurring artifactsmay arise as a function of one or more of the acquisition parameters.

Deblurring system 200 comprises acquisition parameter transform 304,which receives acquisition parameters 302 as input, and maps acquisitionparameters 302 to one or more outputs, wherein the one or more outputsmay be incorporated into input layers 322 of deep neural network 324. Insome embodiments, acquisition parameter transform 304 comprises atrained neural network, which receives one or more acquisitionparameters at an input layer, and maps the one or more acquisitionparameters to a first output comprising a plurality of values, whereinthe plurality of values may be equal in number to a plurality of pixelsor voxels of blurred image 310. In some embodiments, acquisitionparameter transform 304 comprises an analytical mapping, such as anequation or model, which receives one or more acquisition parameters andmaps to the first output. In the embodiment shown in FIG. 2, acquisitionparameter transform 304 may produce an equal number of output values, asthere are pixel or voxel values in blurred image 310, thereby enablingeach of the plurality of pixels or voxels of blurred image 310 to haveat least a single output value from acquisition parameter transform 304concatenated therewith.

Input layers 322 comprise acquisition layers 306 produced from output ofacquisition parameters 304, blurred image 310, and optionally, parametermaps 308, which may include one or more parameter maps of an imagedspace, such as a B₁ ⁺ field, a B₀ field, a T1 map, a T2 map, a gradientnonlinearity map, an encoding error map, a motion map (where themotion/velocity may be estimated or measured), and so on. Input layers322 are received by deep neural network 324, and mapped to deblurredimage 320. Deep neural network 324 comprises learned convolutionalfilters 314 (learned during a training processand learneddeconvolutional filters 318 (learned during a training process). Bypropagating data from input layers 322 through the convolutional anddeconvolutional layers of deep neural network 324, deblurred image 320is produced.

Acquisition parameters 302 comprise one or more settings used by animaging system during acquisition of blurred image 310, and/or one ormore physiological attributes of an imaged tissue/patient and/orenvironmental conditions. For example, if the blurred image 310 is an MRimage, acquisition parameters 302 may comprise one or more of an echotrain length, repetition time, echo time, echo spacing, target flipangle(s), sampling pattern, acquisition order, physiological signals, orother parameters/settings used by an MRI system during acquisition ofblurred image 310, or relating to an imaged patient. Blurred image 310is a medical image of an anatomical region, comprising a plurality ofvalues, wherein each value may be referred to as a pixel (for 2D images)or voxel (for 3D images). Blurred image 310 comprises one or moreblurring artifacts, which may generally be described as aspreading/smearing of intensity values across a larger region of theimage than the originating/underlying anatomical region. Blurring may bemathematically described as a convolutional of an original, unblurredimage, with a point-spread-function, wherein the point-spread-functionacts to spread out point sources of intensity from the unblurred imageto produce regions of intensity occupying a relatively larger region ofthe image. As an example, blurred image 310 comprises an MR image ofhuman abdomen comprising spatially heterogeneous blurring, wherein anextent and/or type of blurring varies based on pixel/voxel withinblurred image 310. That is, blurred image 310 may comprise a firstregion having a first degree of blurring, and a second region comprisinga second degree of blurring, wherein the first and second degrees andtypes of blurring are not the same.

Acquisition parameters 302 may comprise a vector or scalar, and may beinput into acquisition parameter transforms 304. Acquisition parametertransform 304 is configured to map acquisition parameters 302 to one ormore outputs, which may be incorporated into deep neural network 324and/or concatenated with parametric maps 308 and blurred image 310before being input into deep neural network 324. In some embodiments, aplurality of outputs may be produced by a plurality of acquisitionparameter transforms, wherein the acquisition parameter transforms maycomprise functions, neural networks, and/or numerical models. Eachoutput produced by the one or more acquisition parameter transforms maybe incorporated into deep neural network 324. In some embodiments, eachoutput of each the plurality of acquisition parameter transforms may beinput into deep neural network 324 via input layers 322.

Acquisition layers 306, comprise one or more feature maps/layers, eachlayer being of a same dimension as blurred image 310, thus enablingdirect concatenation of the acquisition layers 306 and the blurred image310. In some embodiments, acquisition parameter transform 304 mayreceive an acquisition parameter 302 comprising a single scalar value,and may map the single scalar value to a plurality of values, whereinthe plurality of values is equal in number to a plurality of pixels inblurred image 310. In some embodiments, acquisition parameter transform304 may map a plurality of acquisition parameters 302 to a plurality ofacquisition parameter layers. In one example, if blurred image 310comprises a 512×512×120 voxel volume, acquisition layers 306 maycomprise one or more layers, each with dimensions of 512×512×120,thereby enabling direct concatenation with blurred image 310. By mappingacquisition parameters 302, which may comprise a scalar (a singleacquisition parameter) or a vector (a plurality of acquisitionparameters), to a higher dimensional space, information regardingspatial heterogeneity of blurring within an imaged space may be appendedto blurred image 310 prior to being input into deep neural network 324,thereby providing deep neural network 324 with additional informationregarding the spatial distribution of blurring in blurred image 310,which may arise as a function of the one or more acquisition parameters302.

Parameter map(s) 308 may comprise one or more parameters of an imagedspace, wherein the imaged space corresponds to blurred image 310. Insome embodiments, the parameter map(s) 308 may comprise one or morephysical properties/parameters of the region imaged by blurred image310. In one example, parameter map(s) 308 may comprise a magnetic fieldstrength, a proton density field map, a density field map, a B₁ ⁺ fieldinhomogeneity map, etc.

Input layers 322, comprising acquisition layers 306, parameter maps 308,and blurred image 310, may be propagated through the plurality of layerswithin deep neural network 324, to map intensity values of blurred image310 to intensity values of deblurred image 320. Deep neural network 324comprises learned convolutional filters 314, and learned deconvolutionalfilters 318. Deep neural network 324 may further comprise one or moredensely connected layers (not shown), and one or more pooling layers(not shown), one or more up sampling layers (not shown), and one or moreReLU layers (not shown), or any layers conventional in the art ofmachine learning.

Output of deep neural network 324 may be used to produce deblurred image320, which comprises an image of a same anatomical region as blurredimage 310, but with at least one or more blurring artifacts at leastpartially removed.

Turning to FIG. 3, a schematic of a second embodiment of a deblurringprocess 300 for deblurring a medical image is shown. Elements ofdeblurring process 300 previously introduced and discussed in previousfigures may retain their numbering. Deblurring process 300 may beimplemented by image processing system 100 to at least partially removeone or more blurring artifacts from a blurred medical image. Deblurringsystem 300 serves to illustrate an embodiment of a deblurring system,wherein acquisition parameters 302 may be mapped via acquisitionparameter transforms 304 to one or more values, which may then be usedto set one or more convolutional and/or deconvolutional filter weightsof deep neural network 324. The convolutional and/or deconvolutionalfilter weights set based on output of acquisition parameter transform304 may enable the deep neural network to be informed of thetypes/extent of blurring present in blurred image 310. By applyingconvolution and deconvolution filters specifically selected for a givenset of acquisition parameters, one or more blurring artifacts may bemore consistently and thoroughly removed from blurred image 310. Bymapping acquisition parameters to output, and incorporating the outputinto deep neural network 324, the deep neural network 324 may receive atleast partial information regarding the extent, type, and distributionof blurring artifacts in blurred image 310, as the blurring artifactsmay arise as a function of one or more of the acquisition parameters.

Deblurring system 300 comprises acquisition parameter transforms 304,which receive acquisition parameters 302 as input, and map acquisitionparameters 302 to one or more outputs, wherein the one or more outputsmay be incorporated directly into deep neural network 324 by setting oneor more parameters of deep neural network 324 based the output (see FIG.6 for a more detailed discussion). In some embodiments, acquisitionparameter transform 304 comprises a Fourier transform, and the firstoutput of the acquisition parameter transform 304 comprises apoint-spread-function corresponding to the blurring present in theblurred image 310 acquired with the one or more acquisition parameters302.

Input layers 322 comprise blurred image 310, and optionally, parametermaps 308, which may include one or more parameter maps of an imagedspace, such as a B₁ ⁺ field, a B₀ field, a T1 map, a T2 map, a gradientnonlinearity map, an encoding error map, a motion map (where themotion/velocity may be estimated or measured), and so on. Input layers322 are received by deep neural network 324, and mapped to deblurredimage 320.

Deep neural network 324 comprises convolutional filters 312 (set basedon output from acquisition parameter transforms 304), learnedconvolutional filters 314 (learned during a training process),deconvolutional filters 316 (set based on output from acquisitionparameter transform 304), and learned deconvolutional filters 318(learned during a training process). By propagating data from inputlayers 322 through the convolutional and deconvolutional layers of deepneural network 324, deblurred image 320 is produced.

Acquisition parameters 302 may comprise a vector or scalar, and may beinput into acquisition parameter transforms 304. Acquisition parametertransforms 304 are configured to map acquisition parameters 302 to oneor more outputs, which may be incorporated into deep neural network 324.Output from acquisition parameter transforms 304 may be incorporatedinto deep neural network 324 directly, by setting convolutional filters312 and/or de-convolutional filters 316 based on the output ofacquisition parameter transforms 304. In some embodiments, each outputincorporated into deep neural network 324 may be produced by a separateacquisition parameter transform, such that there is a one to onecorrespondence between outputs and acquisition parameter transforms. Inother embodiments, a single acquisition parameter transform may mapacquisition parameters 302 to a plurality of outputs, wherein theplurality of outputs may each be incorporated into the deep neuralnetwork 324 via a plurality of distinct mechanisms/channels, such asthough described in reference to FIG. 3.

Input layers 322, comprising parameter maps 308, and blurred image 310,may be propagated through the plurality of layers within deep neuralnetwork 324, to map intensity values of blurred image 310 to intensityvalues of deblurred image 320. Deep neural network 324 comprisesconvolutional filters 312, learned convolutional filters 314,deconvolutional filters 316, and learned deconvolutional filters 318.Deep neural network 324 therefore comprises both parameters/layerslearned during training using training data pairs, as well as parametersselected based on output from acquisition parameter transforms 304.

Deep neural network 324 comprises convolutional filters 312, anddeconvolutional filters 316, which may be determined or selected basedon output of acquisition parameter transforms 304. In some embodiments,acquisition parameter transforms 304 map one or more acquisitionparameters to a plurality of values, wherein the plurality of valuescomprise a plurality of weights, and wherein one or more filters withinconvolutional filters 312 and/or deconvolutional filters 316 are setbased on the plurality of weights. In some embodiments, one or moreconvolution filters 312 and/or one or more deconvolution filters 316 areset equal to the output of acquisition parameter transforms 304. In someembodiments, output of acquisition parameter transforms 304 may comprisea plurality of weights representing a point spread function, or aninverse mapping of a point spread function, determined based on one ormore acquisition parameters input into acquisition parameter transforms304.

Deep neural network 324 further comprises learned convolutional filters314 and learned deconvolutional filters 318, which comprise filterslearned during training of deep neural network 324. By including bothparameters learned during a training process, as well as parametersselected/determined based on output from one or more acquisitionparameter transforms 304, deep neural network 324 may be enabled to atleast partially anticipate the types, extent, and/or distribution ofblurring artifacts present in blurred image 310, which may enable moreprecise and consistent mapping of blurred image intensity values todeblurred image intensity values, improving the diagnostic quality ofone or more blurred images, and reducing the need to re-image a patientanatomical region.

In some embodiments, output from the plurality of acquisition parametertransforms may be input into deep neural network 324 both via inputlayers 322 and by setting one or more convolutional filters 312 ordeconvolutional filters 316 based on the output. That is to say,embodiments shown in FIGS. 2 and 3 may be combined. The set ofparameters transformed to be part of the input layer 322 may be the sameas or different than the set of parameters transformed to be the filters312/316. There may or may not be overlap. In some embodiments, theacquisition parameter transform 304 may map acquisition parameters 302to a plurality of outputs, each being incorporated into the deep neuralnetwork 324 via a plurality of distinct mechanisms/channels, such asthose described in more detail in FIGS. 5 and 6.

Turning to FIG. 4, CNN architecture 400 for mapping a blurred medicalimage to a deblurred medical image based on output of an acquisitionparameter transform is shown. CNN architecture 400 provides a moredetailed illustration of a deep neural network, such as deep neuralnetwork 324, which may execute deblurring of a blurred medical image. Insome embodiments, a subset of the parameters of CNN architecture 400 maybe selected/determined based on output of one or more acquisitionparameter transforms, while other parameters of CNN architecture 400 maybe learned using a training algorithm. For example, as indicated indeblurring system 200 shown in FIG. 3, CNN architecture 400 mayincorporate one or more outputs of the acquisition parameter transform.

CNN architecture 400 represents a U-net architecture, which may bedivided into an autoencoder portion (descending portion, elements 402b-430) and an autodecoder portion (ascending portion, elements 432-456a). CNN architecture 400 is configured to receive medical imagesincluding one or more blurring artifacts, which may comprise a magneticresonance (MR) image, computed tomography (CT) image, positron emissiontomography (PET) image, X-ray image, or ultrasound image. In oneembodiment, CNN architecture 400 is configured to receive data from ablurred medical image of an anatomical region, such as blurred medicalimage 402 a, comprising a plurality of pixels/voxels, and map the inputblurred medical image data to a deblurred medical image of the sameanatomical region, such as deblurred medical image 456 b, based onoutput of an acquisition parameter transform. CNN architecture 400comprises a series of mappings, from an input image tile 402 b, whichmay be received by an input layer, through a plurality of feature maps,and finally to an output deblurred medical image 456 b, which may beproduced based on output from output layer 456 a. In some embodiments,CNN architecture 400 is configured to receive an output from a firstacquisition parameter transform in the form of an acquisition parameterlayer and concatenate the acquisition parameter layer data from blurredmedical image 402 a, wherein the concatenated input image data andacquisition parameter layer may be fed into input tile 402 b andpropagated through the layers of CNN architecture 400. In someembodiments, CNN architecture 400 is configured to set one or moreconvolutional filters, and/or one or more deconvolutional filters basedon output of an acquisition parameter transform. CNN architecture 400may be configured to receive a plurality of outputs from a correspondingplurality of acquisition parameter transforms, which may be incorporatedinto CNN architecture 400 according to one or more, of the abovedescribed embodiments.

The various elements comprising CNN architecture 400 are labeled inlegend 458. As indicated by legend 458, CNN architecture 400 includes aplurality of feature maps (and/or copied feature maps) connected by oneor more operations (indicated by arrows). The arrows/operations receiveinput from either an external file, or a previous feature map, andtransform/map the received input to output to produce a next featuremap. Each feature map may comprise a plurality of neurons, where in someembodiments, each neuron may receive input from a subset of neurons of aprevious layer/feature map, and may compute a single output based on thereceived inputs, wherein the output may be propagated/mapped to asubset, or all, of the neurons in a next layer/feature map.

Feature maps may be described using the terms length, width, and depth,wherein each term refers to a number of neurons comprising the featuremap (e.g., how many neurons long, how many neurons wide, and how manyneurons deep, a specified feature map is). Length and width, as used inreference to a feature map, correspond to the spatial dimensions of theimage being processed, and may in some cases correspond to a number ofpixels/voxels of an image. Depth, as used in reference to a feature mapmay correspond to a number of features in each feature channel.

The transformations/mappings performed between each feature map areindicated by arrows, wherein each distinct type of arrow corresponds toa distinct type of transformation, as indicated by legend 458. Rightwardpointing solid black arrows indicate 3×3 convolutions with a stride of1, wherein output from a 3×3 grid of features of an immediatelypreceding feature map (wherein the 3×3 grid extends through all layersof the immediately preceding feature map) are mapped to a singlefeature, at a single depth, of a current feature map by performing a dotproduct between the outputs/activations of the 3×3 grid of featurechannels and a 3×3 filter, (comprising 9 weights for each layer/unit ofdepth of the immediately preceding feature map). In some embodiments,the convolutional filter weights may be selected based on output from anacquisition parameter transform. In some embodiments the convolutionalfilter weights may be learned during a training process. The filtersused to perform the 3×3 convolutions are herein referred to asconvolution filters, convolutional filters, convolution kernels, orconvolutional kernels.

Downward pointing arrows indicate 2×2 max pooling operations, whereinthe max value from a 2×2 grid of feature channels at a single depth ispropagated from an immediately preceding feature map to a single featureat a single depth of a current feature map, thereby resulting in anoutput feature map with a 4-fold reduction in spatial resolution ascompared to the immediately preceding feature map. In one example, maxpooling of a 2×2 grid of activations from an immediately precedingfeature map, wherein the 2×2 grid of activations comprises (2, 1.4, 10,4.4) produces an output of (10), as 10 is the maximum value of theactivations within the 2×2 grid.

Upward pointing arrows indicate 2×2 up-convolutions of stride 2, whichcomprise performing a transpose convolution (also referred to herein asa deconvolution) using a deconvolution filter comprising a plurality ofweights (filters used to perform transpose convolutions are herein alsoreferred to as deconvolutional filters or deconvolution filters) mappingoutput from a single feature channel at each feature depth of animmediately preceding feature map to a 2×2 grid of features at a singlefeature depth in a current feature map, thereby increasing the spatialresolution of the immediately preceding feature map 4-fold.

Rightward pointing dash-tailed arrows indicate copying and cropping of afeature map for concatenation with another, later occurring, featuremap. Cropping enables the dimensions of the copied feature map to matchthe dimensions of the feature map with which the copied feature map isto be concatenated. It will be appreciated that when the size of thefirst feature map being copied and the size of the second feature map tobe concatenated with the first feature map, are equal, no cropping maybe performed.

Rightward pointing arrows with hollow heads indicate a 1×1 convolutionwith stride 1, in which each feature channel in an immediately precedingfeature map is mapped to a single feature channel of a current featuremap, or in other words, wherein a 1-to-1 mapping of feature channelsbetween an immediately preceding feature map and a current feature mapoccurs. Processing at every feature map may include the above-describedconvolutions and deconvolutions, as well as activations, whereactivation functions are non-linear functions that restrict the outputvalues of the processing to a bounded range.

In addition to the operations indicated by the arrows within legend 458,CNN architecture 400 includes solid filled rectangles corresponding tofeature maps, wherein feature maps comprise a height (top to bottomlength as shown in FIG. 4, corresponds to a y spatial dimension in anx-y plane), width (not shown in FIG. 4, assumed equal in magnitude toheight, corresponds to an x spatial dimension in an x-y plane), anddepth (a left-right length as shown in FIG. 4, corresponds to the numberof features within each feature channel). Likewise, CNN architecture 400includes hollow (unfilled) rectangles, corresponding to copied andcropped feature maps, wherein copied feature maps comprise height (topto bottom length as shown in FIG. 4, corresponds to a y spatialdimension in an x-y plane), width (not shown in FIG. 4, assumed equal inmagnitude to height, corresponds to an x spatial dimension in an x-yplane), and depth (a length from a left side to a right side as shown inFIG. 4, corresponds to the number of features within each featurechannel).

Starting at input image tile 402 b (herein also referred to as an inputlayer), data corresponding to a blurred medical image 402 a is input andmapped to a first set of features. In some embodiments, blurred medicalimage 402 a, which may comprise one or more layers corresponding to oneor more features of the image (such as each intensity value of amulti-color image) may further comprise one or more concatenatedacquisition parameter layers, produced by one or more acquisitionparameter transforms. in some embodiments, acquisition parameter layersconcatenated with blurred medical image 402 a may indicate anexpected/anticipated type, or intensity of blurring artifact at eachpixel position of blurred medical image 402 a. Blurred medical image 402a may comprise a two-dimensional (2D) or three-dimensional (3D)image/map of a patient anatomical region. In some embodiments, the inputdata from blurred medical image 402 a is pre-processed (e.g.,normalized) before being processed by the neural network.

Output layer 456 a may comprise an output layer of neurons, wherein eachneuron may correspond to a pixel of a predicted deblurred medical image456 b (or residual), wherein output of each neuron may correspond to thepredicted pixel intensity in specified location within the outputdeblurred medical image 456 b.

In this way, CNN architecture 400 may enable mapping of a plurality ofintensity values from a blurred medical image 402 a to a plurality ofintensity values of a deblurred medical image 456 b, wherein an extentof blurring of one or more blurring artifacts present in blurred medicalimage 402 a is reduced or eliminated in deblurred medical image 456 b.In some embodiments, CNN architecture 400 may enable mapping of one ormore features of a pixel/voxel of a blurred medical image to one or moreproperties deblurred medical image. CNN architecture 400 illustrates thefeature map transformations which occur as an input image tile ispropagated through the neuron layers of a convolutional neural network,to produce a deblurred medical image. In one example, CNN architecture400 may enable mapping of a plurality of pixel/voxel intensity values ofan blurred medical image to a residual map, wherein a deblurred medicalimage may be produced by combining the residual map with the inputblurred medical image 402 a, such as by pixelwise addition of values.

The weights (and biases) of the convolutional layers in CNN architecture400 may be learned during training, and/or incorporated/set based onoutput from one or more acquisition parameter transforms, as will bediscussed in more detail with reference to FIGS. 5, 6, and 7 below. CNNarchitecture 400 may be trained by calculating a difference between apredicted deblurred medical image, and a ground truth deblurred medicalimage, wherein the ground truth deblurred medical image may comprise amedical image without blurring artifacts. The difference between thepredicted deblurred medical image and the ground truth deblurred medicalimage may be used to determine a loss, and the loss may be backpropagated through the neural network to update the weights (and biases)of each feature map using gradient descent, or any other method ofparameter optimization known in the art of machine learning. A pluralityof training data pairs, comprising blurred medical images andcorresponding ground truth deblurred medical images, may be used duringthe training process of CNN architecture 400.

Although not shown in FIG. 4, it will be appreciated that the currentdisclosure encompasses neural network architectures comprising one ormore regularization layers, including batch normalization layers,dropout layers, Gaussian noise layers, and other regularization layersknown in the art of machine learning which may be used during trainingto mitigate overfitting and increase training efficiency while reducingtraining duration.

It should be understood that CNN architecture 400 shown in FIG. 4 is forillustration, not for limitation. Any appropriate neural network can beused herein for predicting a deblurred medical image from a blurredmedical image using output from one or more acquisition parametertransforms, such as ResNet, recurrent neural networks, GeneralRegression Neural Network (GRNN), etc. One or more specific embodimentsof the present disclosure are described above in order to provide athorough understanding. These described embodiments are only examples ofsystems and methods for predicting deblurred medical images from blurredmedical images using a deep neural network and one or more acquisitionparameters and one or more acquisition parameter transforms. The skilledartisan will understand that specific details described in theembodiments can be modified when being placed into practice withoutdeviating the spirit of the present disclosure.

Turning to FIG. 5, acquisition parameter transform 500 is shown.Acquisition parameter transform 500 may be stored in acquisitionparameter transform module 210 within image processing system 31, andmay be used to map one or more acquisition parameters to one or moreacquisition parameter layers, which may be concatenated with data from ablurred medical image and received by a deep neural network for mappingto a deblurred medical image, as illustrated in deblurring system 200 ofFIG. 3. Acquisition parameter transform 500 is configured to map avector of acquisition parameters (1D) to acquisition parameter layer 518having a size and dimension matching the size and dimension of a blurredmedical image. Acquisition parameter layer 518 may then be concatenatedwith data from the blurred medical image, and the combined blurredmedical image data and acquisition parameter layer data may be inputinto a deep neural network, thereby providing the deep neural networkwith additional information regarding acquisition parameters of theblurred medical image, which may thereby inform the deep neural networkof the types of blurring and/or extent of blurring within the blurredmedical image.

Acquisition parameter transform 500 is configured to receive a vector ofn acquisition parameters (a 1D data object), comprising a firstacquisition parameter AP₁ 502, through to an n^(th) acquisitionparameter Acquisition parameter transform 504, where n is a positiveinteger greater than 1. In some embodiments, acquisition parametertransform 500 may be configured to receive a single acquisitionparameter, without deviating from the scope of the current disclosure.Each of the n acquisition parameters includes a corresponding inputnode/neuron in a first/input layer of acquisition parameter transform500, such that a 1-to-1 correspondence between acquisition parametersand input nodes/neurons exists. As shown in acquisition parametertransform 500, the n acquisition parameters are matched by n nodes inthe input layer. Each of node (1,1) 506 through to node (1,n) 508, mayreceive a single acquisition parameter as input, and map the singlereceived acquisition parameter to output by passing the acquisitionparameter through an activation function, wherein the activationfunction may include a bias term.

Output from each of input node (1,1) 506 through to input node (1,n) 508is received by each of first hidden layer node (2,1) 510 through tofirst hidden layer node (2,J) 512, wherein J is a positive integergreater than 1, which may be equal to, or not equal to, n. In otherwords, output from each node of the first layer of acquisition parametertransform 500 is received by each node of the first hidden layer ofacquisition parameter transform 500, and therefore the first hiddenlayer of the acquisition parameter transform 500 may be referred to as afully connected layer. Each node of the first hidden layer ofacquisition parameter transform 500 calculates a dot product usingoutput from each node of the previous layer and each correspondingweight according to the below equation.

$Y_{j} = {f\left( {{\sum\limits_{i = 1}^{n}{W_{ji}X_{i}}} + B_{j}} \right)}$

Where X_(i) is the i-th neuron of the preceding layer, Y_(j) is the j-thneuron of the subsequent layer, W_(ji) is the weight, and B_(j) is thebias. In some embodiments, the activation function ƒ is a rectifiedlinear unit (ReLU) function, for example, plain ReLU function, leakyReLU function, parametric ReLU function, etc.

Acquisition parameter transform 500 may include a positive integernumber of fully connected hidden layers, analogous to the first hiddenlayer described above, and indicated by a horizontal ellipsis betweenthe first hidden layer and the output layer. Acquisition parametertransform 500 may further include one or more dropout layers, or otherregularization layers which may facilitated training and mitigate overfitting of the training data.

Acquisition parameter transform 500 further comprises output nodes (C,1)514 through to output node (C,R) 516, wherein C designates that theoutput layer is the C-th layer of acquisition parameter transform 500,and that the C-th layer includes R nodes, wherein C and R are integersgreater than 1. Each of output nodes (C,1) 514 through to output node(C,R) 516 receives weighted output from each node of an immediatelypreceding hidden layer, and computes an activation therefrom, analogousto the manner in which each hidden layer computes an activation based onthe weighted input from each preceding node/neuron in an immediatelypreceding layer. Each activation value is then multiplied by acorresponding weight, and the result is used to produce acquisitionparameter layer 518.

In the embodiment shown by FIG. 5, each output node includes X by Yweights (wherein the values of each of the X by Y weights may bedifferent for each output node), which are used to map the activationvalue from each output node to a separate depth of acquisition parameterlayer 518. The size of X and Y are chosen based on the size of the inputimage (the blurred medical image) with which acquisition parameter layer518 is to be concatenated. For example, if a blurred medical imagecomprises a 3D image of 512 pixels by 512 pixels by 100 pixels, X may beset to 512, Y may be set to 512, and R may be set to 100.

In one embodiment, output node (C,1) 514, which represents the firstoutput node of the output layer of acquisition parameter transform 500,produces a single output/activation value, and the output is then mappedto a first depth 520 of acquisition parameter layer 518 using X by Yweights. Similarly, output node (C,R) 516, the last node in the outputlayer, produces an activation value, which is propagated to a last,R-th, depth of acquisition parameter layer 518 using X by Y weights. Theblurred medical image with which acquisition parameter layer 518 may beconcatenated is of width X, height Y, and depth R, and thereforerepresents a 3D image, such as a 3D image of a patient anatomicalstructure. As a specific example, if R is equal to 3, X is equal to 10,and Y is equal to 10, the output layer will comprise 3 output nodes, andeach of the 3 output nodes will produce 100 output values (bymultiplying a single activation value for each output node by 100associated weights), wherein a first depth of acquisition parameterlayer 518 comprises the 100 output values produced by output node (C,1),a second depth of acquisition parameter layer 518 (not shown) comprisesthe 100 output values produced by output node (C,2) (not shown), and thethird depth of acquisition parameter layer 518 comprises the 100 outputvalues produced by output node (C,R). In this way, one or moreacquisition parameters may be mapped to a data object of a samedimension and size as a blurred medical image, enabling directconcatenation of the acquisition parameter layer 518 with the blurredmedical image prior to input of the blurred medical image data into adeep neural network.

It will be appreciated that the current disclosure encompassesacquisition parameter transforms with architectures other than thatdepicted in FIG. 5 for mapping one or more acquisition parameters to adata object of a same dimension and size as a blurred medical image. Forexample, each output node may produce values used in each depth ofacquisition parameter layer 518. In another example, each output node ofthe output layer may output a single value to be incorporated intoacquisition parameter layer 518, such that if acquisition parameterlayer 518 comprises 100 total values, for example, the output layercomprises 100 output nodes. In some embodiments, acquisition parameterlayer 518 may be of a same dimension, but of a smaller size than theblurred medical image.

Turning to FIG. 6, acquisition parameter transform 600 is shown.Acquisition parameter transform 600 may be stored in acquisitionparameter transform module 210 within image processing system 31, andmay be used to map one or more acquisition parameters to one or moreconvolutional or deconvolutional filters, which may be incorporated intoone or more convolutional and/or deconvolutional layers of a deep neuralnetwork, respectively. Acquisition parameter transform 600 is configuredto map a vector of acquisition parameters (1D) to a plurality of filterweights, wherein the filter weights may be used in one or moreconvolutional and/or deconvolutional filters of a deep neural network,such as deep neural network 324 shown in FIG. 3, or CNN architecture 400shown in FIG. 4.

Acquisition parameter transform 600 is configured to receive a vector ofn acquisition parameters (a 1D data object), comprising a firstacquisition parameter AP₁ 602, through to an n^(th) acquisitionparameter Acquisition parameter transform 604, where n is a positiveinteger greater than 1. In some embodiments, acquisition parametertransform 600 may be configured to receive a single acquisitionparameter, without deviating from the scope of the current disclosure.Each of the n acquisition parameters includes a corresponding inputnode/neuron in a first/input layer of acquisition parameter transform600, such that a 1-to-1 correspondence between acquisition parametersand input nodes/neurons exists. As shown in acquisition parametertransform 600, the n acquisition parameters are matched by n nodes inthe input layer. Each of node (1,1) 606 through to node (1,n) 608, mayreceive a single acquisition parameter as input, and may map the singlereceived acquisition parameter to output by passing the acquisitionparameter through an activation function, wherein the activationfunction may include a bias term.

Output from each of input node (1,1) 606 through to input node (1,n) 608is received by each of first hidden layer node (2,1) 610 through tofirst hidden layer node (2,J) 612, wherein J is a positive integergreater than 1, which may be equal to, or not equal to, n. In otherwords, output from each node of the first layer of acquisition parametertransform 600 is received by each node of the first hidden layer ofacquisition parameter transform 600, and therefore the first hiddenlayer of the acquisition parameter transform 600 may be referred to as afully connected layer. Each node of the first hidden layer ofacquisition parameter transform 600 calculates a dot product usingoutput from each node of the previous layer and each correspondingweight according to the below equation.

$Y_{j} = {f\left( {{\sum\limits_{i = 1}^{n}{W_{ji}X_{i}}} + B_{j}} \right)}$

Where X_(i) is the i-th neuron of the preceding layer, Y_(j) is the j-thneuron of the subsequent layer, W_(ji) is the weight, and B_(j) is thebias. In some embodiments, the activation function ƒ is a rectifiedlinear unit (ReLU) function, for example, plain ReLU function, leakyReLU function, parametric ReLU function, etc.

Acquisition parameter transform 600 may include a positive integernumber of fully connected hidden layers, analogous to the first hiddenlayer described above, and indicated by a horizontal ellipsis betweenthe first hidden layer and the output layer. Acquisition parametertransform 600 may further include one or more dropout layers, or otherregularization layers which may facilitated training and mitigate overfitting of the training data.

Acquisition parameter transform 600 further comprises first output node(C,1) 614 through to last output node (C,R) 616, wherein C designatesthat the output layer is the C-th layer of acquisition parametertransform 600, and that the C-th layer includes R nodes, wherein C and Rare integers greater than 1, which may or may not be equal to eachother. Each of first output node (C,1) 614 through to last output node(C,R) 616 receives weighted output from each node of an immediatelypreceding hidden layer, and computes an activation value therefrom,analogous to the manner in which each hidden layer computes anactivation based on the weighted input from each node/neuron in animmediately preceding layer. Each activation value is then multiplied bya corresponding weight, and the result is used to produce filter 618,wherein filter 618 comprises a convolutional and/or deconvolutionalfilter having i by R weights (that is, filter 618 comprises i columnsand R rows).

In the embodiment shown by FIG. 6, each of the R output nodes includes iassociated weights, wherein the i associated weights are used to map theactivation value from each output node to a i distinct filter weights,such that each output node produces the filter weights in a single rowof filter 618. It will be appreciated that, although shown as comprisinga single depth, filter 618 may comprise any positive integer number ofdepths, such that filter 618 may comprise i×R×D weights, wherein D isthe depth of the filter. In one embodiment, filter 618 comprises a 3×3convolutional filter with depth of 1, such that the output layer ofacquisition parameter transform 600 comprises 3 output nodes, each with3 distinct weights, mapping the 3 activation values to 9 total filterweights.

Filter 618 may be incorporated into one or more convolutional and/ordeconvolutional layers of a deep neural network, such as deep neuralnetwork 324 of FIG. 3, or CNN architecture 400 of FIG. 4, and may enablethe deep neural network to more consistently deconvolve/deblur a blurredmedical image.

It will be appreciated that the current disclosure encompassesacquisition parameter transforms with architectures other than thatdepicted in FIG. 6 for mapping one or more acquisition parameters toconvolutional and/or deconvolutional filters. For example, each outputnode may produce values used in each depth of filter 618. In anotherexample, each output node of the output layer may output a single valueto be incorporated into filter 618, such that if filter 618 comprises 27total values, for example, the output layer comprises 27 output nodes.

The deep neural network(s) may be trained by using a plurality pairs ofblurred medical images and corresponding sharp (or pristine) images. Insome embodiments, in a sharp-blurred medical image pair, the blurredmedical image is reconstructed from the acquired raw data by a medicaldevice while the sharp image is obtained by processing the blurred imagethrough known denoising/deblurring methods or any combination thereof.In some embodiments, in a sharp-blurred medical image pair, the sharpand blurred images are acquired for the same anatomical region but withdifferent acquisition parameters. The blurred images are used as inputto the deep neural network and the the sharp images are used as thegrouth truth for reference.

A difference between the deblurred medical image output by the deepneural network and the corresponding sharp medical image is determinedand backpropogated through the layers/feature maps of the deep neuralnetwork.

In some embodiments, parameters of the deep neural network and the oneor more acquisition parameter transforms are adjusted in a single phase,wherein the difference between the predicted deblurred medical image andthe ground truth deblurred medical image is used to adjust parametersthroughout both the deep neural network and the acquisition parametertransform in a single phase. Alternatively, the deep neural network andthe one or more acquisition parameter transforms may be trained inalternating phases, wherein during a first phase, parameters of the deepneural network are held fixed, while parameters of the one or moreacquisition parameter transforms is adjusted based on the training data.During a second phase, parameters of the one or more acquisitionparameter transforms may be held fixed while the parameters of the deepneural network are adjusted based on the training data. Alternation oftraining phases may continue until a threshold accuracy of prediction ismet, or until the parameters of the one or more acquisition parametertransforms and the deep neural network have converged (that is, when arate of change of the parameters of the one or more acquisitionparameter transforms and the deep neural network have a rate of a changeper round which is less than a pre-determined threshold rate of change).

Referring to FIG. 7, a flow chart of a method 700 for deblurring ablurred medical image using a deep neural network and one or moreacquisition parameter transforms is shown, according to an exemplaryembodiment. Method 700 may be implemented by the image processing system100, an edge device connected to the imaging device, a cloud incommunication with the imaging device, or any appropriate combinationthereof.

Method 700 begins at operation 702, wherein a blurred medical image isreceived. In some embodiments, the image processing system receives theblurred medical image from an imaging system via communicative coupling,such as over a network. In some embodiments, the image processing systemreceives the blurred medical image from non-transitory memory. Althoughdescribed with reference to a single blurred medical image forsimplicity, it will be appreciated that the current disclosure providesfor mapping a plurality of blurred medical images to a plurality (or toa single) deblurred medical image. For example, a number of input layerscorresponding to a number of blurred medical images may be increased toaccommodate the number of blurred medical images to be deblurred,without deviating from the disclosure herein provided.

At operation 704, one or more acquisition parameters associated with theblurred medical image(s) are received. Acquisition parameters associatedwith, or corresponding to, a blurred medical image may comprise one ormore settings, parameters, or conditions, used or present duringacquisition of the blurred medical image. In some embodiments,acquisition parameters comprise settings of an imaging device usedduring a scan/image acquisition. In some embodiments, acquisitionparameters comprise one or more attributes of the patient anatomicalregion imaged during a scan/image acquisition. Acquisition parametersmay be stored with, or indexed by, the medical image(s) with which theycorrespond, such that rapid and computationally efficient retrieval ofthe one or more acquisition parameters associated with a blurred medicalimage may be enabled.

At operation 706 the received acquisition parameters are input into oneor more trained acquisition parameter transforms which maps theacquisition parameters to a first output. In some embodiments the firstoutput may comprise one or more acquisition parameter layers, which maybe concatenated with the blurred medical image, and input into a deepneural network. In some embodiments the first output may comprise one ormore values, which may be used to set one or more weights in aconvolution filter and/or deconvolution filter of a deep neural network.In some embodiments, the output may be used for both the input layer andthe weights of the filters. In some embodiments, the acquisitionparameter transform comprises an input layer configured to receive theacquisition parameter(s), an output layer configured to produce thefirst output, and at least one fully connected layer between the inputlayer and the output layer. In some embodiments, the blurred medicalimage comprises a number of pixels or voxels, and the first output fromthe acquisition parameter transform comprises a number of output values,wherein the number of output values is evenly divisible by the number ofpixels or voxels, such that an integer number of output values from thefirst output may be concatenated with data from each pixel or voxel ofthe blurred input image. See FIGS. 5 and 6 for a more detaileddescription of the acquisition parameter transforms and the types ofoutput which may be produced therefrom. Although acquisition parametertransforms are herein described as comprising learnable parameters, itwill be appreciated that the current disclosure further provides foracquisition parameter transforms comprising fixed models/simulations,wherein the parameters are fixed, such as according to a physical modelor equation, which may be employed to map one or more acquisitionparameters to a first output having the properties herein described.

At operation 708 the first output from the one or more acquisitionparameter transforms is incorporated into a deep neural network, such asdeep neural network 324, or CNN architecture 400. Incorporation of thefirst output from the one or more acquisition parameter transforms intoa deep neural network may occur as described with reference to FIGS. 3,4, 5, and 6, and as such, will be described briefly here. In someembodiments, incorporating the first output with the deep neural networkcomprises concatenating the first output with a parameter map and theblurred medical image, wherein the parameter map may comprise a map ofone or more physical or physiological properties of an imagedspace/imaged anatomical region, and inputting the first output, theparameter map, and the blurred medical image, into an input layer of thetrained deep neural network. In some embodiments, incorporating thefirst output with the deep neural network comprises concatenating thefirst output with the blurred medical image and inputting both the firstoutput and the blurred medical image into an input layer of the traineddeep neural network. In some embodiments, the first output comprises aplurality of values, and incorporating the first output into the deepneural network comprises setting a plurality of weights in the traineddeep neural network based on the plurality of values, wherein in someembodiments the deep neural network comprises a CNN, and the pluralityof weights comprise a deconvolution filter of a deconvolutional layer ora convolution filter of a convolutional layer of the convolutionalneural network. In some embodiments, the output may be used for both theinput layer and the weights of the filters.

At operation 710, the blurred medical image is mapped to a second outputusing the deep neural network. Mapping the blurred medical image to thesecond output comprises inputting data from the blurred medical image,including any additional concatenated data, into an input layer/inputtile of a deep neural network, and propagating the input data througheach layer of the deep neural network until a second output is producedby an output layer of the deep neural network. In some embodiments, thedeep neural network comprises a convolutional neural network, whereinone or more filters (convolutional or deconvolutional) are set based onthe first output from the one or more acquisition parameter transforms,and the one or more filters are applied to the data from the blurredmedical image as the data propagates through the deep neural network.

At operation 712, a deblurred medical image is generated using thesecond output from the deep neural network. In some embodiments, thesecond output comprises a residual map, and producing the deblurredmedical image from the blurred medical image using the second outputcomprises combining the residual map with the blurred medical image toproduce the deblurred medical image. In other words, the residual mapmay comprises a plurality of values, one or each pixel or voxel of theinput blurred medical image, which describes the intensity differencebetween each pixel or voxel of the blurred image and the intensity ofeach pixel or voxel of a corresponding deblurred medical image.Combining the residual map with the blurred medical image to produce thedeblurred medical image may comprise pixelwise addition of valuesbetween the residual map and the blurred medical image. In someembodiments, the second output from the deep neural network comprises amap of pixel/voxel intensity values of the deblurred medical image.

At operation 714, the image processing system displays the deblurredmedical image via a display device. In some embodiments, a user, via auser input device, may select the deblurred image for further imageprocessing, such as image segmentation, background noise suppression,pathology identification, super resolution, etc. using models trained onsharp/unblurred medical images.

In this way, method 700 enables deblurring of a blurred medical image ina time efficient and more consistent manner, by integrating informationregarding one or more acquisition parameters, using one or moreacquisition parameter transforms, into the deep neural network used toproduce the deblurred medical image. The deblurred medical image may bemore efficiently processed by further downstream image processingmodels, which may have been trained using sharp medical images.

Turning to FIG. 8, one example of a point spread function 804, and acorresponding transverse magnetization signal decay 802 is shown. Thenominal signal decay 802 is fit with a model for a nominal T2 value (orsome fractional combination of T2 values, which may result in amulti-exponential decay) based on one or more acquisition parameters,such as echo spacing, echo train length, sample schedule/order, andfiltering of the raw k-space data. Transforming signal decay 802 fromthe frequency domain to the spatial domain may be accomplished viaFourier-transform, resulting in point spread function 804. In esomeembodiments, inverting signal decay 802 prior to transformation to thespatial domain may be used to produce the weights of one or moreconvolution or deconvolution kernels, which may be used, for example, atlayers 312 and/or 316.

In some embodiments, point spread functions similar to point spreadfunction 804 may be predicted by an acquisition parameter transformbased on one or more acquisition parameters, and inverted, to produce adeconvolution filter which may be incorporated into a deep neuralnetwork and used to deconvolve a blurred medical image. As blurring maybe modeled as a convolution of a point spread function over a sharpimage, deconvolving a blurred image using an inverted point spreadfunction may enable consistent deblurring, particularly when the inversepoint spread function is predicted based on acquisition parameters usedduring acquisition of a blurred medical image to be deblurred, insteadof a generic point spread function.

In some embodiments, point spread function 804 may be produced by ananalytical model as a function of one or more acquisition parameters.

Turning to FIG. 9, examples are shown of blurred medical images andcorresponding deblurred medical images, produced according toembodiments of the present disclosure. Blurred medical image 902,comprises an MR cross sectional image of human abdomen, and includes oneor more blurring artifacts which reduce the sharpness of the variousboundaries of the imaged anatomical regions. Sharp/deblurred medicalimage 904 comprises the same anatomical regions as depicted in blurredmedical image 902, however sharp medical image 904 has been deblurredaccording to a deblurring method herein disclosed, such as method 700.As can be seen, the boundaries of the anatomical regions imaged in sharpmedical image 904 are more sharp/less smeared, than the correspondinganatomical regions imaged in blurred medical image 902, which may enablemore precise analysis and/or diagnosis based on the imaged anatomicalregions. Further, the extent of blurring within blurred medical image902 varies throughout the image, however deblurred medical image 904shows a consistent degree of deblurring throughout.

Blurred medical image 906, comprises an MR cross sectional image of ahuman hip region, and includes one or more blurring artifacts whichreduce the sharpness of the various boundaries of the imaged anatomicalregions. Sharp/deblurred medical image 908 comprises the same anatomicalregions as depicted in blurred medical image 906, however sharp medicalimage 908 has been deblurred according to a deblurring method hereindisclosed, such as method 700. As can be seen, fine structures in thetissue and bone in the anatomical regions captured in deblurred medicalimage 908 are more clearly defined, than the corresponding finestructures in blurred medical image 906. Thus, deblurred medical image908 may enable more precise analysis and/or diagnosis.

Although FIG. 9 provides two specific examples of anatomical regions,imaged via MRI, which may be deblurred using the systems and methodsdisclosed herein, it will be appreciated that the current disclosureprovides for deblurring of substantially any medical images of anyanatomical regions. In one embodiment, a single deep neural network maybe trained using training data pairs of substantially similar anatomicalregions, captured/acquired using a single medical imaging modality, andthe deep neural network may be employed in deblurring of blurred medicalimages of anatomical regions substantially similar to those of thetraining data pairs. In other embodiments, a single deep neural networkmay be trained using training data pairs comprising a plurality ofdistinct medical imaging modalities of distinct anatomical regions,thereby producing a more generalized deep neural network which mayenable deblurring of a wide range of medical images of variousanatomical regions using a single deep neural network.

The technical effect of incorporating one or more acquisition parametersinto a trained deep neural network is that the deep neural network mayreceive at least partial information regarding the type, extent, and/orspatial distribution of blurring in a blurred medical image, enablingthe trained deep neural network to selectively deblur the receivedblurred medical image with a higher degree of consistency.

One or more specific embodiments of the present disclosure are describedabove in order to provide a thorough understanding. These describedembodiments are only examples of systems and methods for selectivelydenoising a medical image by using a deep neural network. The skilledartisan will understand that specific details described in theembodiments can be modified when being placed into practice withoutdeviating the spirit of the present disclosure.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” and “the” are intended to mean thatthere are one or more of the elements. The terms “first,” “second,” andthe like, do not denote any order, quantity, or importance, but ratherare used to distinguish one element from another. The terms“comprising,” “including,” and “having” are intended to be inclusive andmean that there may be additional elements other than the listedelements. As the terms “connected to,” “coupled to,” etc. are usedherein, one object (e.g., a material, element, structure, member, etc.)can be connected to or coupled to another object regardless of whetherthe one object is directly connected or coupled to the other object orwhether there are one or more intervening objects between the one objectand the other object. In addition, it should be understood thatreferences to “one embodiment” or “an embodiment” of the presentdisclosure are not intended to be interpreted as excluding the existenceof additional embodiments that also incorporate the recited features.

In addition to any previously indicated modification, numerous othervariations and alternative arrangements may be devised by those skilledin the art without departing from the spirit and scope of thisdescription, and appended claims are intended to cover suchmodifications and arrangements. Thus, while the information has beendescribed above with particularity and detail in connection with what ispresently deemed to be the most practical and preferred aspects, it willbe apparent to those of ordinary skill in the art that numerousmodifications, including, but not limited to, form, function, manner ofoperation and use may be made without departing from the principles andconcepts set forth herein. Also, as used herein, the examples andembodiments, in all respects, are meant to be illustrative only andshould not be construed to be limiting in any manner.

1. A method for deblurring a medical image comprising: receiving ablurred medical image and one or more acquisition parameterscorresponding to acquisition of the blurred medical image; incorporatingthe one or more acquisition parameters into a trained deep neuralnetwork; and mapping, by the trained deep neural network, the blurredmedical image to a deblurred medical image.
 2. The method of claim 1,wherein incorporating the one or more acquisition parameters into thetrained deep neural network comprises: mapping the one or moreacquisition parameters to one or more weights; and incorporating the oneor more weights into the one or more trained deep neural networks. 3.The method of claim 2, wherein the trained deep neural network comprisesa convolutional neural network, and wherein incorporating the one ormore weights into the trained deep neural network comprises setting oneor more convolutional filter weights and/or one or more deconvolutionalfilter weights based on the one or more weights.
 4. The method of claim1, wherein incorporating the one or more acquisition parameters into thetrained deep neural network comprises: mapping the one or moreacquisition parameters to a plurality of values; concatenating theplurality of values with a plurality of pixel or voxel data of theblurred medical image; and inputting the plurality of values and theplurality of pixel or voxel data of the blurred medical image into aninput layer of the trained deep neural network.
 5. The method of claim1, further comprising training the deep neural networkby using aplurality pairs of sharp medical images and corresponding blurredmedical images.
 6. The method of claim 1, wherein the medical image isone of a magnetic resonance (MR) image, computed tomography (CT) image,positron emission tomography (PET) image, X-ray image, or ultrasoundimage.
 7. The method of claim 1, wherein the one or more acquisitionparameters comprise one or more of echo train length, echo spacing, flipangle, voxel spatial dimension, sampling pattern, acquisition order,acceleration factor, physiological state, image reconstructionparameter, detector size, detector number, source-to-detector distance,collimator geometry, slice thickness, dose, detector type, detectorgeometry, ring radius, positron range, depth/line of response,transducer geometry, central frequency, ringdown, spatial pulse length,focal depth, beam apodization, or side lobe amplitude.
 8. A method forreducing blur in a medical image, the method comprising: receiving ablurred medical image acquired by an imaging system; receiving one ormore acquisition parameters of the imaging system applied duringacquisition of the blurred medical image; mapping the one or moreacquisition parameters to a first output using an acquisition parametertransform; incorporating the first output into a convolutional neuralnetwork (CNN); mapping the blurred medical image to a second outputusing the CNN; and producing a deblurred medical image using the secondoutput.
 9. The method of claim 8, wherein the second output comprises aresidual map, and wherein producing the deblurred medical image usingthe second output comprises combining the residual map with the blurredmedical image to produce the deblurred medical image.
 10. The method ofclaim 8, wherein the first output is one or more weights, and whereinincorporating the first output into the CNN comprises setting one ormore convolutional filter weights and/or one or more deconvolutionalfilter weights based on the one or more weights.
 11. The method of claim10, wherein the acquisition parameter transform comprises an analyticalmodel which maps the one or more acquisition parameters to the one ormore weights.
 12. The method of claim 11, wherein the analytical modelis based on a point-spread-function.
 13. The method of claim 8, whereinthe first output is a plurality of values, and wherein incorporating thefirst output into the CNN comprises conconcatenating the plurality ofvalues with a plurality of pixel or voxel data of the blurred medicalimage as input to the CNN.
 14. The method of claim 13, wherein theacquisition parameter transform comprises another trained deep neuralnetwork that maps the one or more acquisition parameters to theplurality of values.
 15. A system for deblurring a medical image, thesystem comprising: a memory storing a trained deep neural network and anacquisition parameter transform; and a processor communicably coupled tothe memory and configured to: receive a blurred medical image; receiveone or more acquisition parameters, wherein the one or more acquisitionparameters correspond to acquisition of the blurred medical image; mapthe acquisition parameter to a first output using the acquisitionparameter transform; incorporate the first output into the trained deepneural network; map the blurred medical image to a second output usingthe trained deep neural network; and produce a deblurred medical imageusing the second output.
 16. The system of claim 15, wherein the traineddeep neural network is a trained convolutional neural network, whereinthe first output is one or more weights, and wherein the processor isconfigured to incorporate the first output into the trained deep neuralnetwork by setting one or more convolutional filter weights and/or oneor more deconvolutional filter weights of the convolutional neuralnetwork based on the one or more weights.
 17. The system of claim 16,wherein the acquisition parameter transform comprises an analyticalmodel which maps the one or more acquisition parameters to the one ormore weights.
 18. The system of claim 17, wherein the analytical modelis based on a point-spread-function.
 19. The system of claim 15, whereinthe first output is a plurality of values, and wherein the processor isconfigured to incorporate the first output into the trained deep neuralnetwork by conconcatenating the plurality of values with a plurality ofpixel or voxel data of the blurred medical image as input to the traineddeep neural network.
 20. The system of claim 19, wherein the acquisitionparameter transform comprises another trained deep neural network thatmaps the one or more acquisition parameters to the plurality of values.